Compositionality, stability and robustness in probabilistic machine learning

Probability theory plays an integral part in the field of machine learning. Its use has been advocated by many [MacKay, 2002; Jaynes, 2003] as it allows for the quantification of uncertainty and the incorporation of prior knowledge by simply applying the rules of probability [Kolmogorov, 1950]. While probabilistic machine learning has been originally restricted to simple models, the advent of new computational technologies, such as automatic differentiation, and advances in approximate inference, such as Variational Inference [Blei et al., 2017], has made it more viable in complex settings. Despite this progress, there remain many challenges to its application to real-world tasks. Among those are questions about the ability of probabilistic models to model complex tasks and their reliability both in training and in the face of unexpected data perturbation. These three issues can be addressed by examining the three properties of compositionality, stability and robustness in these models. Hence, this thesis explores these three key properties and their application to probabilistic models, while validating their importance on a range of applications. The first contribution in this thesis studies compositionality. Compositionality enables the construction of complex and expressive probabilistic models from simple components. This increases the types of phenomena that one can model and provides the modeller with a wide array of modelling options. This thesis examines this property through the lens of Gaussian processes [Rasmussen and Williams, 2006]. It proposes a generic compositional Gaussian process model to address the problem of multi-task learning in the non-linear setting. Additionally, this thesis contributes two methods addressing the issue of stability. Stability determines the reliability of inference algorithms in the presence of noise. More stable training procedures lead to faster, more reliable inferences, especially for complex models. The two proposed methods aim at stabilising stochastic gradient estimation in Variational Inference using the method of control variates [Owen, 2013]. Finally, the last contribution of this thesis considers robustness. Robust machine learning methods are unaffected by unaccounted-for phenomena in the data. This makes such methods essential in deploying machine learning on real-world datasets. This thesis examines the problem of robust inference in sequential probabilistic models by combining the ideas of Generalised Bayesian Inference [Bissiri et al., 2016] and Sequential Monte Carlo sampling [Doucet and Johansen, 2011]

Generalised Bayesian filtering via sequential Monte Carlo

We introduce a framework for inference in general state-space hidden Markov models (HMMs) under likelihood misspecification. In particular, we leverage the loss-theoretic perspective of Generalized Bayesian Inference (GBI) to define generalised filtering recursions in HMMs, that can tackle the problem of inference under model misspecification. In doing so, we arrive at principled procedures for robust inference against observation contamination by utilising the $\beta$-divergence. Operationalising the proposed framework is made possible via sequential Monte Carlo methods (SMC), where the standard particle methods, and their associated convergence results, are readily adapted to the new setting. We demonstrate our approach to object tracking and Gaussian process regression problems, and observe improved performance over standard filtering algorithms

VarGrad: A Low-Variance Gradient Estimator for Variational Inference

We analyse the properties of an unbiased gradient estimator of the evidence lowerbound (ELBO) for variational inference, based on the score function method withleave-one-out control variates. We show that this gradient estimator can be obtainedusing a new loss, defined as the variance of the log-ratio between the exact posteriorand the variational approximation, which we call thelog-variance loss. Undercertain conditions, the gradient of the log-variance loss equals the gradient of the(negative)ELBO. We show theoretically that this gradient estimator, which we callVarGraddue to its connection to the log-variance loss, exhibits lower variance thanthe score function method in certain settings, and that the leave-one-out controlvariate coefficients are close to the optimal ones. We empirically demonstrate thatVarGrad offers a favourable variance versus computation trade-off compared toother state-of-the-art estimators on a discrete variational autoencoder (VAE

CodeCheck: How do our food choices affect climate change?

Different approaches were proposed to predict the carbon footprint of products from the different datasets provided by CodeCheck. Multivariate linear regression and random forest regression models perform well in predicting carbon footprint, especially when - in addition to the nutrition information - the product categories, learned through Latent Dirichlet Allocation (LDA), were used as extra features in the models. The prediction accuracy of the models that were considered varied across datasets. A potential way to display the footprint estimates in the app was proposed